Distributed processing often involves dividing tasks among various processing nodes. For example, different processing nodes may store different data available for querying and processing, and a request may result in processing performed at the various nodes, with results from the nodes merged and/or selected at other nodes to form a result presented to the user. As another example, dividing computations among processing nodes may decrease the latency of certain computations, for example by breaking up the processing of a large input over a number of processing nodes, or having multiple processing nodes perform the same task and using the results of the first node to finish. Distributed systems often include instructions, in the form of software programs or modules, that control how a particular processing problem is distributed and processed.
For example, many systems use remote procedure calls (RPC) to send processing requests between processing nodes. However, such systems are tied to a particular network communications protocol and to a particular flow due to the communication code existing within the application logic. Such integration results in difficulties with upgrades and portability. Furthermore, such systems lack the flexibility to deal with slow or out-of-service nodes in the distribution system.